Rohit Ravi

In-Context Learning

A new dimension to access content of this website

Left: median trials to acquisition for a meta-trained transformer at trial lengths T of 4, 12 and 36: 5, 5 and 6 at C/T 4.5 and 3, 3 and 3 at C/T 27, so acquisition depends on C/T and not on T. Right: eight hashed write-channel runs (GRU arm, E, F, G, P, Q, R, R2) against three stages, write registers and stays specific, retention reference gate, predictions scored; only R passed the first stage and it failed the second, so no run reached scoring.

Where an inference law lives

Artificial and animal learners compared on one conditioning instrument against one ideal observer. A meta-trained transformer acquires a cue-reward contingency in context with timescale invariance; eight pre-registered runs on a surprise-gated write channel all stopped at a gate, and the last gave an exploratory reason why.

 Paired belief at a single probe cue against delay after the eighth reinforcement, 0.5C to 32C, for a meta-trained transformer. Massed training at C/T 4.5 reads 1.23, 1.40, 2.03 and 2.87; spaced training at C/T 27 reads 1.05, 1.09, 1.21 and 1.42; neither falls below the acquisition asymptote of 1. A hierarchical Bayes observer sits at 1.00 at every delay, and a fixed-parameter Bayes observer collapses to 0.03 at 32C on the massed schedule. Values above 1 are marked as a readout-scale effect.

Where learning lives: prior, inference, write

Every learner is three things: a prior it starts with, an inference it makes in state, and a write that makes the inference permanent. The laws animals and in-context learners share live in the second; the differences live in the first and third. A retention probe on a meta-trained transformer gives the inference-only signature, once its reference is read honestly.